7.0. Reproducibility

What must be recorded to reproduce an agent run?

An agent release is a tuple, not only a Git commit:

A hosted model name does not freeze provider weights or service behavior. Likewise, qwen3:4b is a mutable Ollama tag rather than a content pin. Record the installed ID/digest and model metadata from ollama list/ollama show with evaluation evidence. Reproducibility means enough evidence to explain and compare the run, not byte-identical text forever.

How does MLflow link prompt and model evidence?

mise run eval:mlflow registers the exact INSTRUCTION, initializes a logged model with its prompt URI/version and AGENT_MODEL, evaluates the full conversations, then marks the logged model READY or FAILED.

prompt = mlflow.genai.register_prompt(
    name="ops-copilot-instruction",
    template=INSTRUCTION,
    commit_message="Ops Copilot system instruction",
)
logged_model = mlflow.initialize_logged_model(
    name="ops-copilot",
    params={
        "agent_model": settings.model,
        "prompt_uri": prompt.uri,
        "prompt_version": str(prompt.version),
    },
)

The source also links the evaluation to model_id; the excerpt shows the lineage fields that matter.

How do I know which prompt produced this behavior?

By default every runtime uses the committed INSTRUCTION from agent.py — the prompt is whatever Git says it is at that commit. In the host development/evaluation environment, you can compare behavior against a registered version by setting AGENT_PROMPT_URI:

MLFLOW_TRACKING_URI=http://localhost:5000
AGENT_PROMPT_URI=prompts:/ops-copilot-instruction/2

_instruction() then loads that exact version from the self-hosted MLflow prompt registry at startup, so traces and evaluations from that host process are attributable to one immutable prompt text. Configuration validation rejects a malformed URI at startup, and mlflow is imported lazily from the dev dependency group.

The production agent image intentionally installs no dev dependencies, so it does not support AGENT_PROMPT_URI; containers and Kubernetes use the committed instruction. This keeps the runtime image small and avoids an MLflow availability dependency on startup. Promote or roll back production behavior by shipping the evaluated Git/image version, not by setting a registry URI in the deployment.

The full versioning workflow uses the evaluation script rather than a separate registration step, because register_prompt versions by content — an unchanged instruction reuses the existing version, a changed one becomes the next:

1. Edit INSTRUCTION in agent.py (for example, tighten an operating rule).
2. Run mise run eval:mlflow; it registers the edited text as ops-copilot-instruction version 2 and evaluates it in a run named eval-prompt-v2.
3. Every run carries prompt_name and prompt_version tags, so in the MLflow UI you filter or sort the experiment by tags.prompt_version and compare v1 and v2 scorer results side by side.
4. Decide with evidence: keep the new version if the deterministic scorers hold and any judge evidence improves, otherwise roll back.

The registry is the self-hosted MLflow server only; there is no hosted prompt service in this course.

How do you test a prior prompt version?

Point a host development/evaluation process at the prior registered version:

MLFLOW_TRACKING_URI=http://localhost:5000
AGENT_PROMPT_URI=prompts:/ops-copilot-instruction/1

Use that comparison to decide which committed version to ship. Production rollback means redeploying the previously evaluated code/image (or reverting INSTRUCTION, re-evaluating, and releasing a corrected image), then leaving AGENT_PROMPT_URI unset. A pinned host process depends on the MLflow server at startup; the production path deliberately avoids that coupling.

Evolve the eval set with the prompt, not after it. A prompt change that adds behavior (say, a new escalation rule) needs cases that exercise it, or v1 and v2 will score identically while behaving differently. Record the dataset commit next to the prompt version, as in the reproducibility checkpoint: comparing eval-prompt-v1 with eval-prompt-v2 is only valid on the same eval set commit. If the dataset changed in between, re-run the old prompt version against the new dataset before drawing a conclusion.

How do you select the MLflow destination?

MLFLOW_TRACKING_URI=http://localhost:5000
MLFLOW_EXPERIMENT_NAME=ops-copilot
mise run eval:mlflow

The script prints the authoritative tracking URI. It suggests mlflow ui --backend-store-uri ... only when the URI is a local SQLite store, never for a remote HTTP server.

How is data reset for comparison?

cd agents/python
mise run data:reset

That deletes runtime sessions/tasks and the writable incident copy, then the next run initializes from the committed seed. Record the seed Git commit and do not compare a fresh run with one whose mock actions changed service/incident state.

What is deterministic and what is not?

Deterministic: types, validators, SQL seed, retrieval ranking, tool functions, policy callbacks, graph topology, scanner/test configuration, and exact expected trajectories.

Non-deterministic/external: model output, provider implementation, sampling, network timing, Spot scheduling, and live service availability. Pin/control what you can, record what you cannot, and use distributions/evaluations instead of promising replay identity.

What is the reproducibility checkpoint?

Before comparing two runs, record git rev-parse HEAD, AGENT_MODEL_PROVIDER, AGENT_MODEL, OPENAI_BASE_URL, the Ollama model ID when local, gateway profile, prompt URI/version, dataset commit, image tag/digest, and MLflow run/model id. Reset state and use the same eval set; otherwise label the comparison invalid.